Should We Redesign Forecasting Competitions? 1.4. Independent Validation on a Common Data Base

The M3-Competition continues to improve the design of forecasting competitions: It examines more series than any previous competition, improves error analyses. and includes commercial forecasting programs as competitors. To judge where to go from here, I step back to look at the M-Competitions as a whole. I discuss the advantages of the M-Competitions in hopes that they will be retained, describe how to gain additional benefit from future competitions, and finally, describe a low-cost approach to competitions. The M-Competitions provide a model for conducting scientific research. They employ at least five key aspects: empirical testing, multiple hypotheses, large samples, independent validation, and full disclosure. While these aspects might seem obvious, studies in management science seldom include all of them. Empirical testing is necessary to test forecasting methods. Despite the resistance of time-series researchers (Fildes & Makridakis, 1995), interest in empirical studies has been growing among forecasters. Forecasting journals now publish many empirical comparisons. The M-Competitions have led the way in such comparisons. Academic researchers rely heavily upon advocacy (Armstrong, Brodie, & Parsons, 2001b); they develop what they believe to be the beat method (hypothesis), then seek information to support it. It is uncommon in management science for a researcher to examine competing hypotheses. However, nearly 60% of empirical papers in the Journal of Forecasting and the International Journal of Forecasting tested competing hypotheses (Armstrong, 1989). The M-Competitions have been exemplary in providing open calls, thus allowing those with different approaches to participate. Testing should be done from large samples. However, many academic studies, including those in forecasting, do not use large samples. You need only to pick up the latest copies of journals to observe this. The M-Competitions were a departure from this norm. The Brat forecasting competition (Makridakis & Hibon, 1979) examined 111 series (considered large at the time) and the M-Competition (Makridakis et al., 1982) examined 1001. The methods in the competitions were tested on a common holdout database by a researcher who examined the accuracy of forecasts submitted by the competitors. This testing procedure avoided problems inherent in drawing conclusions from prior research in which databases are different. Full disclosure is important to allow others to conduct replications and extensions. Despite a consensus among researchers that replication is vital in advancing scientific knowledge, the number of published replications in the management sciences is negligible, and there are few